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11.2.3 Probability Distributions 


State Probability Distributions: 


Consider a Markov chain {X n , n = 0,1,2,...}, where X n G 5 = {l,2 > .-.,r}. Suppose that we know the 
probability distribution of Xq. More specifically, define the row vector 7r(°) as 

7r(°) = [P(X 0 = l) P(X o = 2) ••• P(Xq = r) ] ■ 

How can we obtain the probability distribution of Xi , X 2 , • • •? We can use the law of total probability. More specifically, 
for any j G S, we can write 

r 

P(X ! =j)=J2 = ^ x ° = k 1 p ( x ° = ^ 

k =1 
r 

= y^PkjP(x 0 = /c). 

it=i 

If we generally define 

7rW = [P(X„ = l) P(X n = 2) ••• P(X n = r)], 

we can rewrite the above result in the form of matrix multiplication 

7T^ = 7r(°)p, 

where P is the state transition matrix. Similarly, we can write 

7r< 2 > = = 7t(°)P 2 . 


More generally, we can write 

tt(" + i) = 7 r Wp, forn = 0 , 1 , 2 ,---; 
ttW = 7 r (»)pn ) forn = 0 , 1 , 2 ,- 


Example 11.5 

Consider a system that can be in one of two possible states, s = { 0 , 1 }. In particular, suppose that the transition matrix is 
given by 


P = 


"1 

2 

1 

. 3 


11 


tJ 


Suppose that the system is in state 0 at time n = 0, i.e., Xq = 0- 


a. Draw the state transition diagram. 

b. Find the probability that the system is in state 1 at time n = 3. 


• Solution 

° a. The state transition diagram is shown in Figure 11.8. 
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3 

Figure 11.8 - A state transition diagram. 

b. Here, we know 

7r(°) = [P(Xo = 0) P(X o = l)] 

= [1 o]. 

Thus, 

7f( 3 ) = 7 t(°)P 3 

1 II 3 

2 2 

I I 

3 3 . 

[ 29 431 

L 72 72 J • 

Thus, the probability that the system is in state 1 at time 71 = 3 is 4|-. 


= [1 0 ] 


72-Step Transition Probabilities: 

Consider a Markov chain {X n , Tl = 0,1,2,...}, where X n E S. If Xq = i, then X\ = j with probability p^. 

That is, pij gives us the probability of going from state i to state j in one step. Now suppose that we are interested in finding 
the probability of going from state i to state j in two steps, i.e., 

pfj = P(X 2 = j\x 0 = i). 

We can find this probability by applying the law of total probability. In particular, we argue that Xi can take one of the 
possible values in S. Thus, we can write 

pjf = P(X 2 = j\X 0 = 0 = = k ’ X o= i)P(X 1 = ^1^0 = i) 

keS 

= P(X 2 = j\X\ = k)P(X i = k\Xo = i ) (by Markov property) 

keS 

— ^ j PkjPik • 

keS 

We conclude 

pfj = P ( X 2 = j\X 0 =i) = PikPkj (H-4) 

keS 


We can explain the above formula as follows. In order to get to state j, we need to pass through some intermediate state k. 

( 2 ) 

The probability of this event is PikPkj ■ To obtain p-j , we sum over all possible intermediate states. Accordingly, we can 
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define the two-step transition matrix as follows: 


" ( 2 ) ( 2 ) 

Pll Pl2 

( 2 ) ( 2 ) 

P21 P22 


( 2 ) ( 2 ) 
-Pr 1 Pr2 


Looking at Eq uation 11.4 . we notice that p i ■ 7 is in fact the element in the ith row and Jth column of the matix 


Pll Pl2 
P21 P22 


Pll Pl2 
P21 P22 


P 2 = 


-Prl Pr2 


Prr J \-Prl Pr2 


Thus, we conclude that the two-step transition matrix can be obtained by squaring the state transition matrix, i.e., 

p( 2 ) _ pi 


More generally, we can define the 71-step transition probabilities p\- ' as 


Pii = P(X n = j\X Q = i), for n = 0,1,2, • • •, 


(11.5) 


and the n-step transition matrix, P ( n ), as 


p( n ) _ 


( n) 

Pll 

(n) 

Pl2 ■ 

(n) 

•• Plr 

( n) 

P21 

(n) 

P22 ■ 

( n) 

•• P2r 

(n) 

Prl 

(n) 

Pr 2 ■ 

(n) 

• • Prr 


We can now generalize Eq uation 11.5 . Let m and Tl be two positive integers and assume -Xo = i. In order to get to state j 
in ( m + n) steps, the chain will be at some intermediate state k after m steps. To obtain , we sum over all 

possible intermediate states: 

(m+n) — r>/ v _ Y_ _ 


= P(X m+n = j\X 0 = i) 

E (m) (n) 

Pik Pkj • 


The above equation is called the Chapman-Kolmogorov equation. Similar to the case of two-step transition probabilities, we 
can show that = P n , for n = 1, 2, 3, • • •. 

The Chapman-Kolmogorov equation can be written as 
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p% +n) = P(X m+n = j\X 0 = i) 

E (m) (n) 

Pik Pkj ■ 

keS 

The 7l-step transition matrix is given by 

p(n) = pn ) for n = 1,2,3, •••. 


<— previous 
next —> 
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